A Proposal for Evaluating Psychosocial Trauma.

BACKGROUND: The objective of the study was to design and validate the Psychosocial Trauma Scale (ETAPS) for assessing psychosocial consequences of collective violence. This instrument proposed the following dimensions: Pre-traumatic Situation, Destruction of Fundamental Beliefs, Intergroup Emotions, and Family and Community Destruction. METHOD: A total of 382 people participated who had been affected by political violence: civil war in El Salvador, forced displacement from Colombia and state violence from Chile. The study had three phases: (1) content validity of the items evaluated by experts; (2) exploratory factor analysis to study the structure of ETAPS, reducing the number of items; (3) convergent (post-traumatic stress symptomatology) and divergent (psychological and social well-being) validity. RESULTS: The EFA showed that ETAPS had a slightly different internal structure from that proposed. The dimensions found were Pre-traumatic Situation and Intergroup Emotions along with two new emerging dimensions: Destruction of Sociality and Personal and Collective Self-Efficacy. Divergent and convergent validity gave expected results except for the pre-traumatic situation. CONCLUSIONS: The ETAPS dimensions show that the effects of violence are broader than the symptoms measured by clinical scales. An instrument with adequate psychometric properties was obtained which will be useful for future studies in the area.

A Proposal for Evaluating Psychosocial Trauma / El Trauma Psicosocial: una Propuesta Para su Evaluación

Background: The objective of the study was to design and validate the Psychosocial Trauma Scale (ETAPS) for assessing psychosocial consequences of collective violence. This instrument proposed the following dimensions: Pre-traumatic Situation, Destruction of Fundamental Beliefs, Intergroup Emotions, and Family and Community Destruction. Method: A total of 382 people participated who had been affected by political violence: civil war in El Salvador, forced displacement from Colombia and state violence from Chile. The study had three phases: (1) content validity of the items evaluated by experts; (2) exploratory factor analysis to study the structure of ETAPS, reducing the number of items; (3) convergent (post-traumatic stress symptomatology) and divergent (psychological and social well-being) validity. Results: The EFA showed that ETAPS had a slightly different internal structure from that proposed. The dimensions found were Pre-traumatic Situation and Intergroup Emotions along with two new emerging dimensions: Destruction of Sociality and Personal and Collective Self-Efficacy. Divergent and convergent validity gave expected results except for the pre-traumatic situation. Conclusions: The ETAPS dimensions show that the effects of violence are broader than the symptoms measured by clinical scales. An instrument with adequate psychometric properties was obtained which will be useful for future studies in the area. (AU)

Antecedentes: el objetivo del estudio fue el diseño y validación de la Escala de Trauma Psicosocial (ETAPS) para evaluar consecuencias psicosociales de la violencia colectiva. Este instrumento propuso las dimensiones: Situación Pre-traumática, Destrucción de Creencias Fundamentales, Emociones Intergrupales, y Destrucción Familiar y Comunitaria. Método: participaron 382 personas afectadas por violencia política: guerra civil en El Salvador, desplazamiento forzado de Colombia y violencia estatal de Chile. El estudio contempló tres fases: (1) validez de contenido de los ítems evaluado por expertos; (2) análisis factorial exploratorio para estudiar la estructura de ETAPS, reduciendo el número de ítems; (3) validez convergente (sintomatología de estrés postraumático) y divergente (bienestar psicológico y social). Resultados: el AFE mostró que la ETAPS tenía una estructura interna que difería parcialmente de la propuesta. Las dimensiones encontradas fueron: Situación Pre-traumática y Emociones Intergrupales, junto con dos nuevas dimensiones: Destrucción de la socialidad y Autoeficacia personal y colectiva. La validez divergente y convergente muestra resultados esperados salvo respecto a la situación pretraumática. Conclusiones: las dimensiones de ETAPS plantean que los efectos de la violencia son más amplios que los síntomas medidos por escalas clínicas. Se obtuvo un instrumento con adecuadas propiedades psicométricas útil para futuros estudios en el área. (AU)

Cheating on Unproctored Internet Test Applications: An Analysis of a Verification Test in a Real Personnel Selection Context.

This study analyses the extent to which cheating occurs in a real selection setting. A two-stage, unproctored and proctored, test administration was considered. Test score inconsistencies were concluded by applying a verification test (Guo and Drasgow Z-test). An initial simulation study showed that the Z-test has adequate Type I error and power rates in the specific selection settings explored. A second study applied the Z-test statistic verification procedure to a sample of 954 employment candidates. Additional external evidence based on item time response to the verification items was gathered. The results revealed a good performance of the Z-test statistic and a relatively low, but non-negligible, number of suspected cheaters that showed higher distorted ability estimates. The study with real data provided additional information on the presence of suspected cheating in unproctored applications and the viability of using item response times as an additional evidence of cheating. In the verification test, suspected cheaters spent 5.78 seconds per item more than expected considering the item difficulty and their assumed ability in the unproctored stage. We found that the percentage of suspected cheaters in the empirical study could be estimated at 13.84%. In summary, the study provides evidence of the usefulness of the Z-test in the detection of cheating in a specific setting, in which a computerized adaptive test for assessing English grammar knowledge was used for personnel selection.

Assessing the Big Five with bifactor computerized adaptive testing.

Multidimensional computerized adaptive testing based on the bifactor model (MCAT-B) can provide efficient assessments of multifaceted constructs. In this study, MCAT-B was compared with a short fixed-length scale and computerized adaptive testing based on unidimensional (UCAT) and multidimensional (correlated-factors) models (MCAT) to measure the Big Five model of personality. The sample comprised 826 respondents who completed a pool with 360 personality items measuring the Big Five domains and facets. The dimensionality of the Big Five domains was also tested. With only 12 items per domain, the MCAT and MCAT-B procedures were more efficient to assess highly multidimensional constructs (e.g., Agreeableness), whereas no differences were found with UCAT and the short scale with traits that were essentially unidimensional (e.g., Extraversion). Furthermore, the study showed that MCAT and MCAT-B provide better content-balance of the pool because, for each Big Five domain, items from all the facets are selected in similar proportions. (PsycINFO Database Record (c) 2018 APA, all rights reserved).

Cheating on unproctored internet test applications: an analysis of a verification test in a Real personnel selection context

This study analyses the extent to which cheating occurs in a real selection setting. A two-stage, unproctored and proctored, test administration was considered. Test score inconsistencies were concluded by applying a verification test (Guo and Drasgow Z-test). An initial simulation study showed that the Z-test has adequate Type I error and power rates in the specific selection settings explored. A second study applied the Z-test statistic verification procedure to a sample of 954 employment candidates. Additional external evidence based on item time response to the verification items was gathered. The results revealed a good performance of the Z-test statistic and a relatively low, but non-negligible, number of suspected cheaters that showed higher distorted ability estimates. The study with real data provided additional information on the presence of suspected cheating in unproctored applications and the viability of using item response times as an additional evidence of cheating. In the verification test, suspected cheaters spent 5.78 seconds per item more than expected considering the item difficulty and their assumed ability in the unproctored stage. We found that the percentage of suspected cheaters in the empirical study could be estimated at 13.84%. In summary, the study provides evidence of the usefulness of the Z-test in the detection of cheating in a specific setting, in which a computerized adaptive test for assessing English grammar knowledge was used for personnel selection

No disponible

Calibrating a new item pool to adptivel assess the big five / Nuevo banco de ítems para evaluar adaptativamente los Cinco Grandes

Background: Even though the Five Factor Model (FFM) has been the dominant paradigm in personality research for the past two decades, very few studies have measured the FFM adaptively. Thus, the purpose of this research was the building of a new item pool to develop a computerized adaptive test (CAT) for personality assessment. Method: A pool of 480 items that measured the FFM facets was developed and applied to 826 participants. Facets were calibrated separately and item selection was performed being mindful of the preservation of unidimensionality of each facet. Then, a post-hoc simulation study was carried out to test the performance of separate CATs to measure the facets. Results: The final item pool was composed of 360 items with good psychometric properties. Findings reveal that a CAT administration of four items per facet (total length of 120 items) provides accurate facets scores, while maintaining the factor structure of the FFM. Conclusions: An item pool with good psychometric properties was obtained and a CAT simulation study demonstrated that the FFM facets could be measured with precision using a third of the items in the pool (AU)

Antecedentes: a pesar de que el Modelo de los Cinco Factores (MCF) ha sido el paradigma predominante durante las últimas dos décadas, muy pocos estudios han medido el MCF de forma adaptativa. El objetivo de esta investigación fue construir un nuevo banco de ítems para desarrollar un test adaptativo informatizado (TAI) para evaluar la personalidad. Método: se desarrolló un banco de 480 ítems para evaluar las facetas del MCF y se aplicó a 826 participantes. Cada faceta se calibró por separado y la selección de ítems se realizó atendiendo a que cada faceta fuese unidimensional. Después se realizó un estudio de simulación post-hoc para evaluar la eficiencia de TAIs a nivel de facetas. Resultados: el banco final estaba formado por 360 ítems con buenas propiedades psicométricas. Los resultados demostraron que la aplicación adaptativa de cuatro ítems por faceta proporciona puntuaciones precisas en las mismas, al mismo tiempo que se mantiene la estructura factorial del MCF. Conclusiones: el banco final está formado por ítems con buenas propiedades psicométricas. La aplicación adaptativa del banco permite medir la personalidad de forma eficiente a nivel de facetas utilizando una tercera parte de los ítems (AU)

Calibrating a new item pool to adaptively assess the Big Five.

BACKGROUND: Even though the Five Factor Model (FFM) has been the dominant paradigm in personality research for the past two decades, very few studies have measured the FFM adaptively. Thus, the purpose of this research was the building of a new item pool to develop a computerized adaptive test (CAT) for personality assessment. METHOD: A pool of 480 items that measured the FFM facets was developed and applied to 826 participants. Facets were calibrated separately and item selection was performed being mindful of the preservation of unidimensionality of each facet. Then, a post-hoc simulation study was carried out to test the performance of separate CATs to measure the facets. RESULTS: The final item pool was composed of 360 items with good psychometric properties. Findings reveal that a CAT administration of four items per facet (total length of 120 items) provides accurate facets scores, while maintaining the factor structure of the FFM. CONCLUSIONS: An item pool with good psychometric properties was obtained and a CAT simulation study demonstrated that the FFM facets could be measured with precision using a third of the items in the pool.

Structural brain connectivity and cognitive ability differences: A multivariate distance matrix regression analysis.

Neuroimaging research involves analyses of huge amounts of biological data that might or might not be related with cognition. This relationship is usually approached using univariate methods, and, therefore, correction methods are mandatory for reducing false positives. Nevertheless, the probability of false negatives is also increased. Multivariate frameworks have been proposed for helping to alleviate this balance. Here we apply multivariate distance matrix regression for the simultaneous analysis of biological and cognitive data, namely, structural connections among 82 brain regions and several latent factors estimating cognitive performance. We tested whether cognitive differences predict distances among individuals regarding their connectivity pattern. Beginning with 3,321 connections among regions, the 36 edges better predicted by the individuals' cognitive scores were selected. Cognitive scores were related to connectivity distances in both the full (3,321) and reduced (36) connectivity patterns. The selected edges connect regions distributed across the entire brain and the network defined by these edges supports high-order cognitive processes such as (a) (fluid) executive control, (b) (crystallized) recognition, learning, and language processing, and (c) visuospatial processing. This multivariate study suggests that one widespread, but limited number, of regions in the human brain, supports high-level cognitive ability differences. Hum Brain Mapp 38:803-816, 2017. © 2016 Wiley Periodicals, Inc.

Inferential Item-Fit Evaluation in Cognitive Diagnosis Modeling.

Research related to the fit evaluation at the item level involving cognitive diagnosis models (CDMs) has been scarce. According to the parsimony principle, balancing goodness of fit against model complexity is necessary. General CDMs require a larger sample size to be estimated reliably, and can lead to worse attribute classification accuracy than the appropriate reduced models when the sample size is small and the item quality is poor, which is typically the case in many empirical applications. The main purpose of this study was to systematically examine the statistical properties of four inferential item-fit statistics: S-X2 , the likelihood ratio (LR) test, the Wald (W) test, and the Lagrange multiplier (LM) test. To evaluate the performance of the statistics, a comprehensive set of factors, namely, sample size, correlational structure, test length, item quality, and generating model, is systematically manipulated using Monte Carlo methods. Results show that the S-X2 statistic has unacceptable power. Type I error and power comparisons favor LR and W tests over the LM test. However, all the statistics are highly affected by the item quality. With a few exceptions, their performance is only acceptable when the item quality is high. In some cases, this effect can be ameliorated by an increase in sample size and test length. This implies that using the above statistics to assess item fit in practical settings when the item quality is low remains a challenge.

A multistage addaptive test of fluid intelligence / Un test adaptativo multietapa de inteligencia fluida

BACKGROUND: Multistage adaptive testing has recently emerged as an alternative to the computerized adaptive test. The current study details a new multistage test to assess fluid intelligence. METHOD: An item pool of progressive matrices with constructed response format was developed, and divided into six subtests. The subtests were applied to a sample of 724 college students and their psychometric properties were studied (i.e., reliability, dimensionality and validity evidence). The item pool was calibrated under the graded response model, and two multistage structures were developed, based on the automatic test assembly principles. Finally, the test information provided by each structure was compared in order to select the most appropriate one. RESULTS: The item pool showed adequate psychometric properties. From the two compared multistage structures, the simplest structure (i.e., routing test and two modules in the next stages) were more informative across the latent trait continuum and were therefore kept. DISCUSSION: Taken together, the results of the two studies support the application of the FIMT (Fluid Intelligence Multistage Test), a multistage test to assess fluid intelligence accurately and innovatively

ANTECEDENTES: los test adaptativos multietapa han emergido recientemente como una alternativa a los test adaptativos informatizados. Se presenta en este estudio un test multietapa para evaluar la inteligencia fluida. MÉTODO: se desarrolló un banco de ítems de matrices progresivas con formato de respuesta construida que posteriormente fue dividido en seis subtests. Los ítems se administraron a un total de 724 estudiantes universitarios. Se estudiaron las propiedades psicométricas de los subtests (fiabilidad, dimensionalidad, evidencias de validez) y se calibró el banco con el modelo de respuesta graduada. Se construyeron después dos estructuras multietapa a través del ensamblaje automático de tests y se comparó la información proporcionada por cada una de ellas. RESULTADOS: los ítems mostraron unas propiedades psicométricas adecuadas. De las dos estructuras puestas a prueba, se conservó finalmente la estructura sencilla, pues resultó más informativa. DISCUSIÓN: los resultados de estos dos estudios avalan el empleo del FIMT, una herramienta que emplea este formato para evaluar de forma innovadora y precisa la inteligencia fluida

A multistage adaptive test of fluid intelligence.

BACKGROUND: Multistage adaptive testing has recently emerged as an alternative to the computerized adaptive test. The current study details a new multistage test to assess fluid intelligence. METHOD: An item pool of progressive matrices with constructed response format was developed, and divided into six subtests. The subtests were applied to a sample of 724 college students and their psychometric properties were studied (i.e., reliability, dimensionality and validity evidence). The item pool was calibrated under the graded response model, and two multistage structures were developed, based on the automatic test assembly principles. Finally, the test information provided by each structure was compared in order to select the most appropriate one. RESULTS: The item pool showed adequate psychometric properties. From the two compared multistage structures, the simplest structure (i.e., routing test and two modules in the next stages) were more informative across the latent trait continuum and were therefore kept. DISCUSSION: Taken together, the results of the two studies support the application of the FIMT (Fluid Intelligence Multistage Test), a multistage test to assess fluid intelligence accurately and innovatively.

Optimal number of strata for the stratified methods in computerized adaptive testing

Test security can be a major problem in computerized adaptive testing, as examinees can share information about the items they receive. Of the different item selection rules proposed to alleviate this risk, stratified methods are among those that have received most attention. In these methods, only low discriminative items can be presented at the beginning of the test and the mean information of the items increases as the test goes on. To do so, the item bank must be divided into several strata according to the information of the items. To date, there is no clear guidance about the optimal number of strata into which the item bank should be split. In this study, we will simulate conditions with different numbers of strata, from 1 (no stratification) to a number of strata equal to test length (maximum level of stratification) while manipulating the maximum exposure rate that no item should surpass (rmax) in its whole domain. In this way, we can plot the relation between test security and accuracy, making it possible to determine the number of strata that leads to better security while holding constant measurement accuracy. Our data indicates that the best option is to stratify into as many strata as possible (AU)

No disponible

Application of cognitive diagnosis models to competency-based situational judgment test / Aplicación de los modelos de diagnóstico cognitivo a tests de juicio situacional basados en competencias

BACKGROUND: Profiling of jobs in terms of competency requirements has increasingly been applied in many organizational settings. Testing these competencies through situational judgment tests (SJTs) leads to validity problems because it is not usually clear which constructs SJTs measure. The primary purpose of this paper is to evaluate whether the application of cognitive diagnosis models (CDM) to competency-based SJTs can ascertain the underlying competencies measured by the items, and whether these competencies can be estimated precisely. METHOD: The generalized deterministic inputs, noisy "and" gate (G-DINA) model was applied to 26 situational judgment items measuring professional competencies based on the great eight model. These items were applied to 485 employees of a Spanish financial company. The fit of the model to the data and the convergent validity between the estimated competencies and personality dimensions were examined. RESULTS: The G-DINA showed a good fit to the data and the estimated competency factors, adapting and coping and interacting and presenting were positively related to emotional stability and extraversion, respectively. CONCLUSIONS: This work indicates that CDM can be a useful tool when measuring professional competencies through SJTs. CDM can clarify the competencies being measured and provide precise estimates of these competencies

ANTECEDENTES: muchas organizaciones definen sus puestos de trabajo en base a las competencias profesionales que requieren. La medición de tales competencias mediante tests de juicio situacional (TJS) presenta problemas de validez, en tanto no suele estar claro los constructos que miden. El objetivo principal de este estudio es evaluar si la aplicación de los modelos de diagnóstico cognitivo (MDC) a estos tests permite clarificar y estimar de forma precisa las competencias medidas. MÉTODO: se aplicó el modelo G-DINA (generalized deterministic inputs, noisy "and" gate) a 26 ítems de juicio situacional que medían competencias profesionales fundamentadas en el modelo great eight. Se aplicó el test a 485 trabajadores de una entidad financiera española. Se examinó el ajuste del modelo a los datos, y la validez convergente entre las competencias estimadas y dimensiones de personalidad. RESULTADOS: G-DINA mostró un buen ajuste a los datos, y los factores competenciales estimados adaptarse y aguantar, e interactuar y presentar mostraron una relación positiva con estabilidad emocional y extraversión, respectivamente. CONCLUSIONES: este trabajo muestra que los MDC pueden ser una herramienta útil para la medición de competencias profesionales a través de TJS, aclarando las competencias que miden y obteniendo estimaciones precisas de las mismas

Application of cognitive diagnosis models to competency-based situational judgment tests.

BACKGROUND: Profiling of jobs in terms of competency requirements has increasingly been applied in many organizational settings. Testing these competencies through situational judgment tests (SJTs) leads to validity problems because it is not usually clear which constructs SJTs measure. The primary purpose of this paper is to evaluate whether the application of cognitive diagnosis models (CDM) to competency-based SJTs can ascertain the underlying competencies measured by the items, and whether these competencies can be estimated precisely. METHOD: The generalized deterministic inputs, noisy "and" gate (G-DINA) model was applied to 26 situational judgment items measuring professional competencies based on the great eight model. These items were applied to 485 employees of a Spanish financial company. The fit of the model to the data and the convergent validity between the estimated competencies and personality dimensions were examined. RESULTS: The G-DINA showed a good fit to the data and the estimated competency factors, adapting and coping and interacting and presenting were positively related to emotional stability and extraversion, respectively. CONCLUSIONS: This work indicates that CDM can be a useful tool when measuring professional competencies through SJTs. CDM can clarify the competencies being measured and provide precise estimates of these competencies.

Exploratory factor analysis in validation studies: uses and recommendations.

BACKGROUND: The Exploratory Factor Analysis (EFA) procedure is one of the most commonly used in social and behavioral sciences. However, it is also one of the most criticized due to the poor management researchers usually display. The main goal is to examine the relationship between practices usually considered more appropriate and actual decisions made by researchers. METHOD: The use of exploratory factor analysis is examined in 117 papers published between 2011 and 2012 in 3 Spanish psychological journals with the highest impact within the previous five years. RESULTS: RESULTS show significant rates of questionable decisions in conducting EFA, based on unjustified or mistaken decisions regarding the method of extraction, retention, and rotation of factors. CONCLUSIONS: Overall, the current review provides support for some improvement guidelines regarding how to apply and report an EFA.

Optimal number of strata for the stratified methods in computerized adaptive testing.

Test security can be a major problem in computerized adaptive testing, as examinees can share information about the items they receive. Of the different item selection rules proposed to alleviate this risk, stratified methods are among those that have received most attention. In these methods, only low discriminative items can be presented at the beginning of the test and the mean information of the items increases as the test goes on. To do so, the item bank must be divided into several strata according to the information of the items. To date, there is no clear guidance about the optimal number of strata into which the item bank should be split. In this study, we will simulate conditions with different numbers of strata, from 1 (no stratification) to a number of strata equal to test length (maximum level of stratification) while manipulating the maximum exposure rate that no item should surpass (r max ) in its whole domain. In this way, we can plot the relation between test security and accuracy, making it possible to determine the number of strata that leads to better security while holding constant measurement accuracy. Our data indicates that the best option is to stratify into as many strata as possible.

A new IRT-based standard setting method: Application to eCat-Listening / Un nuevo método de standard setting basado en la TRI: aplicación a eCat-Listening

Background: Criterion-referenced interpretations of tests are highly necessary, which usually involves the difficult task of establishing cut scores. Contrasting with other Item Response Theory (IRT)-based standard setting methods, a non-judgmental approach is proposed in this study, in which Item Characteristic Curve (ICC) transformations lead to the final cut scores. Method: eCat-Listening, a computerized adaptive test for the evaluation of English Listening, was administered to 1,576 participants, and the proposed standard setting method was applied to classify them into the performance standards of the Common European Framework of Reference for Languages (CEFR). Results: The results showed a classification closely related to relevant external measures of the English language domain, according to the CEFR. Conclusions: It is concluded that the proposed method is a practical and valid standard setting alternative for IRT-based tests interpretations (AU)

Antecedentes: las interpretaciones de los tests referidas a criterio son muy necesarias, lo cual normalmente implica la difícil tarea de establecer puntos de corte. En contraste con otros métodos de standard setting basados en la Teoría de la Respuesta al Ítem (TRI), en este estudio se propone una aproximación no basada en juicios, en que transformaciones de las Curvas Características de los Ítems (CCIs) dan lugar a los puntos de corte finales. Método: se administró eCat-Listening, un test adaptativo informatizado de evaluación de la comprensión oral del inglés, a 1.576 participantes y se aplicó el método de standard setting propuesto para clasificarles en los estándares de ejecución del Marco Común Europeo de Referencia para las lenguas (MCER). Resultados: los resultados mostraron una clasificación estrechamente relacionada con variables externas relevantes sobre dominio del inglés, de acuerdo con el MCER. Conclusiones: se concluye que el método de standard setting propuesto es una alternativa práctica y válida para las interpretaciones de tests basados en TRI (AU)

A new IRT-based standard setting method: application to eCat-listening.

BACKGROUND: Criterion-referenced interpretations of tests are highly necessary, which usually involves the difficult task of establishing cut scores. Contrasting with other Item Response Theory (IRT)-based standard setting methods, a non-judgmental approach is proposed in this study, in which Item Characteristic Curve (ICC) transformations lead to the final cut scores. METHOD: eCat-Listening, a computerized adaptive test for the evaluation of English Listening, was administered to 1,576 participants, and the proposed standard setting method was applied to classify them into the performance standards of the Common European Framework of Reference for Languages (CEFR). RESULTS: The results showed a classification closely related to relevant external measures of the English language domain, according to the CEFR. CONCLUSIONS: It is concluded that the proposed method is a practical and valid standard setting alternative for IRT-based tests interpretations.

Computerized adaptive testing: the capitalization on chance problem.

This paper describes several simulation studies that examine the effects of capitalization on chance in the selection of items and the ability estimation in CAT, employing the 3-parameter logistic model. In order to generate different estimation errors for the item parameters, the calibration sample size was manipulated (N = 500, 1000 and 2000 subjects) as was the ratio of item bank size to test length (banks of 197 and 788 items, test lengths of 20 and 40 items), both in a CAT and in a random test. Results show that capitalization on chance is particularly serious in CAT, as revealed by the large positive bias found in the small sample calibration conditions. For broad ranges of theta, the overestimation of the precision (asymptotic Se) reaches levels of 40%, something that does not occur with the RMSE (theta). The problem is greater as the item bank size to test length ratio increases. Potential solutions were tested in a second study, where two exposure control methods were incorporated into the item selection algorithm. Some alternative solutions are discussed.

Computerized adaptive testing: the capitalization on chance problem

This paper describes several simulation studies that examine the effects of capitalization on chance in the selection of items and the ability estimation in CAT, employing the 3-parameter logistic model. In order to generate different estimation errors for the item parameters, the calibration sample size was manipulated (N = 500, 1000 and 2000 subjects) as was the ratio of item bank size to test length (banks of 197 and 788 items, test lengths of 20 and 40 items), both in a CAT and in a random test. Results show that capitalization on chance is particularly serious in CAT, as revealed by the large positive bias found in the small sample calibration conditions. For broad ranges of Delta, the overestimation of the precision (asymptotic Se) reaches levels of 40%, something that does not occur with the RMSE (θ). The problem is greater as the item bank size to test length ratio increases. Potential solutions were tested in a second study, where two exposure control methods were incorporated into the item selection algorithm. Some alternative solutions are discussed (AU)

Se describen varios estudios de simulación para examinar los efectos de la capitalización del azar en la selección de ítems y la estimación de rasgo en Tests Adaptativos Informatizados (TAI), empleando el modelo logístico de 3 parámetros. Para generar diferentes errores de estimación de los parámetros de los ítems, se manipuló el tamaño de la muestra de calibración (N = 500, 1000 y 2000 sujetos), así como la ratio entre tamaño del banco y longitud del test (bancos de 197 y 788 ítems, longitudes del test de 20 y 40 ítems), ambos tanto en un TAI como en un test aleatorio. Los resultados muestran que la capitalización del azar es especialmente importante en el TAI, donde se obtuvo un sesgo positivo en las condiciones de escaso tamaño de la muestra. Para rangos amplios de θ, la sobrestimación de la precisión (Se asintótico) alcanza niveles del 40%, algo que no ocurre con los valores de RMSE (Delta). El problema es mayor a medida que se incrementa la ratio entre el tamaño del banco de ítems y la longitud del test. Varias soluciones fueron puestas a prueba en un segundo estudio, donde se incorporaron dos métodos para el control de la exposición en los algoritmos de selección de los ítems. Se discuten también algunas soluciones alternativas (AU)